We want to take the to-do item input from the user and send it to the server, so that we can save it somehow and display it back to her later.
As I started writing this chapter, I immediately skipped to what I thought was the right design: multiple models for lists and list items, a bunch of different URLs for adding new lists and items, three new view functions, and about half a dozen new unit tests for all of the above. But I stopped myself. Although I was pretty sure I was smart enough to handle all those problems at once, the point of TDD is to allow you to do one thing at a time, when you need to. So I decided to be deliberately short-sighted, and at any given moment only do what was necessary to get the functional tests a little further.
It’s a demonstration of how TDD can support an iterative style of development—it may not be the quickest route, but you do get there in the end. There’s a neat side benefit, which is that it allows me to introduce new concepts like models, dealing with POST requests, Django template tags, and so on one at a time rather than having to dump them on you all at once.
None of this says that you shouldn’t try and think ahead, and be clever. In the next chapter we’ll use a bit more design and up-front thinking, and show how that fits in with TDD. But for now let’s plough on mindlessly and just do what the tests tell us to.
At the end of the last chapter, the tests were telling us we weren’t able to save the user’s input. For now, we’ll use a standard HTML POST request. A little boring, but also nice and easy to deliver—we can use all sorts of sexy HTML5 and JavaScript later in the book.
To get our browser to send a POST request, we need to do two things:
Give the <input> element a name= attribute
Wrap it in a <form> tag with method="POST".
Let’s adjust our template at lists/templates/home.html:
In [1]:
%cd ../testing/superlists/
In [5]:
%%writefile lists/templates/home.html
<html>
<head>
<title>To-Do lists</title>
</head>
<body>
<h1>Your To-Do list</h1>
<form method="POST">
<input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
</form>
<table id="id_list_table">
</table>
</body>
</html>
Now, running our FTs gives us a slightly cryptic, unexpected error:
In [6]:
!python functional_tests.py
When a functional test fails with an unexpected failure, there are several things we can do to debug them:
We’ll look at all of these over the course of this book, but the time.sleep option is one I find myself using very often. Let’s try it now. We add the sleep just before the error occurs:
functional_tests.py.
# When she hits enter, the page updates, and now the page lists
# "1: Buy peacock feathers" as an item in a to-do list table
inputbox.send_keys(Keys.ENTER)
import time
time.sleep(10)
table = self.browser.find_element_by_id('id_list_table')
Depending on how fast Selenium runs on your PC, you may have caught a glimpse of this already, but when we run the functional tests again, we’ve got time to see what’s going on: you should see a page that looks like Figure 5-1, with lots of Django debug information.
If you’ve never heard of a Cross-Site Request Forgery exploit, why not look it up now? Like all security exploits, it’s entertaining to read about, being an ingenious use of a system in unexpected ways…
When I went back to university to get my Computer Science degree, I signed up for the Security module out of a sense of duty: Oh well, it’ll probably be very dry and boring, but I suppose I’d better take it. It turned out to be one of the most fascinating modules of the whole course—absolutely full of the joy of hacking, of the particular mindset it takes to think about how systems can be used in unintended ways.
I want to recommend the textbook for my course, Ross Anderson’s Security Engineering. It’s quite light on pure crypto, but it’s absolutely full of interesting discussions of unexpected topics like lock-picking, forging bank notes, inkjet printer cartridge economics, and spoofing South African Air Force jets with replay attacks. It’s a huge tome, about three inches thick, and I promise you it’s an absolute page-turner.
Django’s CSRF protection involves placing a little auto-generated token into each generated form, to be able to identify POST requests as having come from the original site. So far our template has been pure HTML, and in this step we make the first use of Django’s template magic. To add the CSRF token we use a template tag, which has the curly-bracket/percent syntax, {% … %}—famous for being the world’s most annoying two-key touch-typing combination:
lists/templates/home.html.
In [3]:
%%writefile lists/templates/home.html
<html>
<head>
<title>To-Do lists</title>
</head>
<body>
<h1>Your To-Do list</h1>
<form method="POST">
<input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
{% csrf_token %}
</form>
<table id="id_list_table">
</table>
</body>
</html>
Django will substitute that during rendering with an <input type="hidden"> containing the CSRF token. Rerunning the functional test will now give us an expected failure:
In [6]:
!python3 functional_tests.py
Since our time.sleep is still there, the test will pause on the final screen, showing us that the new item text disappears after the form is submitted, and the page refreshes to show an empty form again. That’s because we haven’t wired up our server to deal with the POST request yet—it just ignores it and displays the normal home page.
We can remove the time.sleep now though:
functional_tests.py.
# "1: Buy peacock feathers" as an item in a to-do list table
inputbox.send_keys(Keys.ENTER)
table = self.browser.find_element_by_id('id_list_table')
Because we haven’t specified an action= attribute in the form, it is submitting back to the same URL it was rendered from by default (ie, /), which is dealt with by our home_page function. Let’s adapt the view to be able to deal with a POST request.
That means a new unit test for the home_page view. Open up lists/tests.py, and add a new method to HomePageTest—I copied the previous method, then adapted it to add our POST request and check that the returned HTML will have the new item text in it:
lists/tests.py (ch05l005).
def test_home_page_returns_correct_html(self):
[...]
def test_home_page_can_save_a_POST_request(self):
request = HttpRequest()
request.method = 'POST'
request.POST['item_text'] = 'A new list item'
response = home_page(request)
self.assertIn('A new list item', response.content.decode())Are you wondering about the line spacing in the test? I’m grouping together three lines at the beginning which set up the test, one line in the middle which actually calls the function under test, and the assertions at the end. This isn’t obligatory, but it does help see the structure of the test. Setup, Exercise, Assert is the typical structure for a unit test. You can see that we’re using a couple of special attributes of the HttpRequest: .method and .POST (they’re fairly self-explanatory, although now might be a good time for a peek at the Django request and response documentation). Then we check that the text from our POST request ends up in the rendered HTML. That gives us our expected fail:
$ python3 manage.py test
[...]
AssertionError: 'A new list item' not found in '<html> [...]We can get the test to pass by adding an if and providing a different code path for POST requests. In typical TDD style, we start with a deliberately silly return value:
lists/views.py.
from django.http import HttpResponse
from django.shortcuts import render
def home_page(request):
if request.method == 'POST':
return HttpResponse(request.POST['item_text'])
return render(request, 'home.html')
That gets our unit tests passing, but it’s not really what we want. What we really want to do is add the POST submission to the table in the home page template.
Passing Python Variables to Be Rendered in the Template We’ve already had a hint of it, and now it’s time to start to get to know the real power of the Django template syntax, which is to pass variables from our Python view code into HTML templates.
Let’s start by seeing how the template syntax lets us include a Python object in our template. The notation is {{ ... }}, which displays the object as a string:
lists/templates/home.html.
<body>
<h1>Your To-Do list</h1>
<form method="POST">
<input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
{% csrf_token %}
</form>
<table id="id_list_table">
<tr><td>{{ new_item_text }}</td></tr>
</table>
</body>
How can we test that our view is passing in the correct value for new_item_text? How do we pass a variable to a template? We can find out by actually doing it in the unit test—we’ve already used the render_to_string function in a previous unit test to manually render a template and compare it with the HTML the view returns. Now let’s add the variable we want to pass in:
lists/tests.py.
self.assertIn('A new list item', response.content.decode())
expected_html = render_to_string(
'home.html',
{'new_item_text': 'A new list item'}
)
self.assertEqual(response.content.decode(), expected_html)
As you can see, the render_to_string function takes, as its second parameter, a mapping of variable names to values. We’re giving the template a variable named new_item_text, whose value is the expected item text from our POST request.
When we run the unit test, render_to_string will substitute {{ new_item_text }} for A new list item inside the <td>. That’s something the actual view isn’t doing yet, so we should see a test failure:
self.assertEqual(response.content.decode(), expected_html)
AssertionError: 'A new list item' != '<html>\n <head>\n [...]Good, our deliberately silly return value is now no longer fooling our tests, so we are allowed to rewrite our view, and tell it to pass the POST parameter to the template:
lists/views.py (ch05l009).
def home_page(request):
return render(request, 'home.html', {
'new_item_text': request.POST['item_text'],
})
Running the unit tests again:
ERROR: test_home_page_returns_correct_html (lists.tests.HomePageTest)
[...]
'new_item_text': request.POST['item_text'],
KeyError: 'item_text'An unexpected failure.
If you remember the rules for reading tracebacks, you’ll spot that it’s actually a failure in a different test. We got the actual test we were working on to pass, but the unit tests have picked up an unexpected consequence, a regression: we broke the code path where there is no POST request.
This is the whole point of having tests. Yes, we could have predicted this would happen, but imagine if we’d been having a bad day or weren’t paying attention: our tests have just saved us from accidentally breaking our application, and, because we’re using TDD, we found out immediately. We didn’t have to wait for a QA team, or switch to a web browser and click through our site manually, and we can get on with fixing it straight away. Here’s how:
lists/views.py.
def home_page(request):
return render(request, 'home.html', {
'new_item_text': request.POST.get('item_text', ''),
})
Look up dict.get if you’re not sure what’s going on there.
The unit tests should now pass. Let’s see what the functional tests say:
AssertionError: False is not true : New to-do item did not appear in table Hmm, not a wonderfully helpful error. Let’s use another of our FT debugging techniques: improving the error message. This is probably the most constructive technique, because those improved error messages stay around to help debug any future errors:
functional_tests.py.
self.assertTrue(
any(row.text == '1: Buy peacock feathers' for row in rows),
"New to-do item did not appear in table -- its text was:\n%s" % (
table.text,
)
)
That gives us a more helpful error message:
AssertionError: False is not true : New to-do item did not appear in table --
its text was:
Buy peacock feathersYou know what could be even better than that? Making that assertion a bit less clever. As you may remember, I was very pleased with myself for using the any function, but one of my Early Release readers (thanks Jason!) suggested a much simpler implementation. We can replace all six lines of the assertTrue with a single assertIn:
functional_tests.py.
self.assertIn('1: Buy peacock feathers', [row.text for row in rows])
Much better. You should always be very worried whenever you think you’re being clever, because what you’re probably being is overcomplicated. And we get the error message for free:
self.assertIn('1: Buy peacock feathers', [row.text for row in rows])
AssertionError: '1: Buy peacock feathers' not found in ['Buy peacock feathers']Consider me suitably chastened.
If, instead, your FT seems to be saying the table is empty ("not found in []"), check your tag — does it have the correct name="item_text" attribute? Without it, the user’s input won’t be associated with the right key in request.POST. The point is that the FT wants us to enumerate list items with a "1:" at the beginning of the first list item. The fastest way to get that to pass is with a quick "cheating" change to the template:
lists/templates/home.html.
<tr><td>1: {{ new_item_text }}</td></tr>
The unit-test/code cycle is sometimes taught as Red, Green, Refactor:
So what do we do during the Refactor stage? What justifies moving from an implementation where we "cheat" to one we’re happy with?
One methodology is eliminate duplication: if your test uses a magic constant (like the "1:" in front of our list item), and your application code also uses it, that counts as duplication, so it justifies refactoring. Removing the magic constant from the application code usually means you have to stop cheating.
I find that leaves things a little too vague, so I usually like to use a second technique, which is called triangulation: if your tests let you get away with writing "cheating" code that you’re not happy with, like returning a magic constant, write another test that forces you to write some better code. That’s what we’re doing when we extend the FT to check that we get a "2:" when inputting a second list item. Now we get to the self.fail('Finish the test!'). If we extend our FT to check for adding a second item to the table (copy and paste is our friend), we begin to see that our first cut solution really isn’t going to, um, cut it:
functional_tests.py.
# There is still a text box inviting her to add another item. She
# enters "Use peacock feathers to make a fly" (Edith is very
# methodical)
inputbox = self.browser.find_element_by_id('id_new_item')
inputbox.send_keys('Use peacock feathers to make a fly')
inputbox.send_keys(Keys.ENTER)
# The page updates again, and now shows both items on her list
table = self.browser.find_element_by_id('id_list_table')
rows = table.find_elements_by_tag_name('tr')
self.assertIn('1: Buy peacock feathers', [row.text for row in rows])
self.assertIn(
'2: Use peacock feathers to make a fly' ,
[row.text for row in rows]
)
# Edith wonders whether the site will remember her list. Then she sees
# that the site has generated a unique URL for her -- there is some
# explanatory text to that effect.
self.fail('Finish the test!')
# She visits that URL - her to-do list is still there.
Sure enough, the functional tests return an error:
AssertionError: '1: Buy peacock feathers' not found in ['1: Use peacock
feathers to make a fly']Before we go further—we’ve got a bad code smell[7] in this FT. We have three almost identical code blocks checking for new items in the list table. There’s a principle called don’t repeat yourself (DRY), which we like to apply by following the mantra three strikes and refactor. You can copy and paste code once, and it may be premature to try and remove the duplication it causes, but once you get three occurrences, it’s time to remove duplication.
We start by committing what we have so far. Even though we know our site has a major flaw—it can only handle one list item—it’s still further ahead than it was. We may have to rewrite it all, and we may not, but the rule is that before you do any refactoring, always do a commit:
$ git diff
# should show changes to functional_tests.py, home.html,
# tests.py and views.py
$ git commit -aBack to our functional test refactor: we could use an inline function, but that upsets the flow of the test slightly. Let’s use a helper method—remember, only methods that begin with test_ will get run as tests, so you can use other methods for your own purposes:
functional_tests.py.
def tearDown(self):
self.browser.quit()
def check_for_row_in_list_table(self, row_text):
table = self.browser.find_element_by_id('id_list_table')
rows = table.find_elements_by_tag_name('tr')
self.assertIn(row_text, [row.text for row in rows])
def test_can_start_a_list_and_retrieve_it_later(self):
[...]
I like to put helper methods near the top of the class, between the tearDown and the first test. Let’s use it in the FT:
functional_tests.py.
# When she hits enter, the page updates, and now the page lists
# "1: Buy peacock feathers" as an item in a to-do list table
inputbox.send_keys(Keys.ENTER)
self.check_for_row_in_list_table('1: Buy peacock feathers')
# There is still a text box inviting her to add another item. She
# enters "Use peacock feathers to make a fly" (Edith is very
# methodical)
inputbox = self.browser.find_element_by_id('id_new_item')
inputbox.send_keys('Use peacock feathers to make a fly')
inputbox.send_keys(Keys.ENTER)
# The page updates again, and now shows both items on her list
self.check_for_row_in_list_table('1: Buy peacock feathers')
self.check_for_row_in_list_table('2: Use peacock feathers to make a fly')
# Edith wonders whether the site will remember her list. Then she sees
[...]
We run the FT again to check that it still behaves in the same way…
AssertionError: '1: Buy peacock feathers' not found in ['1: Use peacock
feathers to make a fly']Good. Now we can commit the FT refactor as its own small, atomic change:
$ git diff # check the changes to functional_tests.py
$ git commit -aAnd back to work. If we’re ever going to handle more than one list item, we’re going to need some kind of persistence, and databases are a stalwart solution in this area.
An Object-Relational Mapper (ORM) is a layer of abstraction for data stored in a database with tables, rows, and columns. It lets us work with databases using familiar object-oriented metaphors which work well with code. Classes map to database tables, attributes map to columns, and an individual instance of the class represents a row of data in the database.
Django comes with an excellent ORM, and writing a unit test that uses it is actually an excellent way of learning it, since it exercises code by specifying how we want it to work.
Let’s create a new class in lists/tests.py:
lists/tests.py.
from lists.models import Item
[...]
class ItemModelTest(TestCase):
def test_saving_and_retrieving_items(self):
first_item = Item()
first_item.text = 'The first (ever) list item'
first_item.save()
second_item = Item()
second_item.text = 'Item the second'
second_item.save()
saved_items = Item.objects.all()
self.assertEqual(saved_items.count(), 2)
first_saved_item = saved_items[0]
second_saved_item = saved_items[1]
self.assertEqual(first_saved_item.text, 'The first (ever) list item')
self.assertEqual(second_saved_item.text, 'Item the second')
You can see that creating a new record in the database is a relatively simple matter of creating an object, assigning some attributes, and calling a .save() function. Django also gives us an API for querying the database via a class attribute, .objects, and we use the simplest possible query, .all(), which retrieves all the records for that table. The results are returned as a list-like object called a QuerySet, from which we can extract individual objects, and also call further functions, like .count(). We then check the objects as saved to the database, to check whether the right information was saved.
Django’s ORM has many other helpful and intuitive features; this might be a good time to skim through the Django tutorial, which has an excellent intro to them.
I’ve written this unit test in a very verbose style, as a way of introducing the Django ORM. You can actually write a much shorter test for a model class, which we’ll see later on, in Chapter 11.
Purists will tell you that a "real" unit test should never touch the database, and that the test I’ve just written should be more properly called an integrated test, because it doesn’t only test our code, but also relies on an external system, ie a database.
It’s OK to ignore this distinction for now—we have two types of test, the high-level functional tests which test the application from the user’s point of view, and these lower-level tests which test it from the programmer’s point of view.
We’ll come back to this and talk about unit tests and integrated tests in Chapter 19, towards the end of the book. Let’s try running the unit test. Here comes another unit-test/code cycle:
ImportError: cannot import name 'Item'Very well, let’s give it something to import from lists/models.py. We’re feeling confident so we’ll skip the Item = None step, and go straight to creating a class:
lists/models.py.
from django.db import models
class Item(object):
pass
That gets our test as far as:
first_item.save()AttributeError: 'Item' object has no attribute 'save'To give our Item class a save method, and to make it into a real Django model, we make it inherit from the Model class:
lists/models.py.
from django.db import models
class Item(models.Model):
pass
The next thing that happens is a database error:
django.db.utils.OperationalError: no such table: lists_itemIn Django, the ORM’s job is to model the database, but there’s a second system that’s in charge of actually building the database called migrations. Its job is to give you the ability to add and remove tables and columns, based on changes you make to your models.py files.
One way to think of it is as a version control system for your database. As we’ll see later, it comes in particularly useful when we need to upgrade a database that’s deployed on a live server.
For now all we need to know is how to build our first database migration, which we do using the makemigrations command:
$ python3 manage.py makemigrations
Migrations for 'lists':
0001_initial.py:
- Create model Item
$ ls lists/migrations
0001_initial.py __init__.py __pycache__If you’re curious, you can go and take a look in the migrations file, and you’ll see it’s a representation of our additions to models.py.
In the meantime, we should find our tests get a little further.
The test actually gets surprisingly far:
$ python3 manage.py test lists
[...]
self.assertEqual(first_saved_item.text, 'The first (ever) list item')
AttributeError: 'Item' object has no attribute 'text'That’s a full eight lines later than the last failure—we’ve been all the way through saving the two Items, we’ve checked they’re saved in the database, but Django just doesn’t seem to have remembered the .text attribute.
Incidentally, if you’re new to Python, you might have been surprised we were allowed to assign the .text attribute at all. In something like Java, that would probably give you a compilation error. Python is more relaxed about things like that.
Classes that inherit from models.Model map to tables in the database. By default they get an auto-generated id attribute, which will be a primary key column in the database, but you have to define any other columns you want explicitly. Here’s how we set up a text field:
lists/models.py.
class Item(models.Model):
text = models.TextField()
Django has many other field types, like IntegerField, CharField, DateField, and so on. I’ve chosen TextField rather than CharField because the latter requires a length restriction, which seems arbitrary at this point. You can read more on field types in the Django tutorial and in the documentation.
Running the tests gives us another database error:
django.db.utils.OperationalError: no such column: lists_item.textIt’s because we’ve added another new field to our database, which means we need to create another migration. Nice of our tests to let us know!
Let’s try it:
$ python3 manage.py makemigrations
You are trying to add a non-nullable field 'text' to item without a default; we
can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows)
2) Quit, and let me add a default in models.py
Select an option:2Ah. It won’t let us add the column without a default value. Let’s pick option 2 and set a default in models.py. I think you’ll find the syntax reasonably self-explanatory:
lists/models.py.
class Item(models.Model):
text = models.TextField(default='')
And now the migration should complete:
$ python3 manage.py makemigrations
Migrations for 'lists':
0002_item_text.py:
- Add field text to itemSo, two new lines in models.py, two database migrations, and as a result, the .text attribute on our model objects is now recognised as a special attribute, so it does get saved to the database, and the tests pass…
$ python3 manage.py test lists
[...]
Ran 4 tests in 0.010s
OKSo let’s do a commit for our first ever model!
$ git status # see tests.py, models.py, and 2 untracked migrations
$ git diff # review changes to tests.py and models.py
$ git add lists
$ git commit -m "Model for list Items and associated migration"Let’s adjust the test for our home page POST request, and say we want the view to save a new item to the database instead of just passing it through to its response. We can do that by adding three new lines to the existing test called test_home_page_cansave a_POST_request:
lists/tests.py.
def test_home_page_can_save_a_POST_request(self):
request = HttpRequest()
request.method = 'POST'
request.POST['item_text'] = 'A new list item'
response = home_page(request)
self.assertEqual(Item.objects.count(), 1) #1
new_item = Item.objects.first() #2
self.assertEqual(new_item.text, 'A new list item') #3
self.assertIn('A new list item', response.content.decode())
expected_html = render_to_string(
'home.html',
{'new_item_text': 'A new list item'}
)
self.assertEqual(response.content.decode(), expected_html)
1
We check that one new Item has been saved to the database. objects.count() is a shorthand for objects.all().count().
2
objects.first() is the same as doing objects.all()[0].
3
We check that the item’s text is correct.
This test is getting a little long-winded. It seems to be testing lots of different things. That’s another code smell—a long unit test either needs to be broken into two, or it may be an indication that the thing you’re testing is too complicated. Let’s add that to a little to-do list of our own, perhaps on a piece of scrap paper:
Writing it down on a scratchpad like this reassures us that we won’t forget, so we are comfortable getting back to what we were working on. We rerun the tests and see an expected failure:
self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1Let’s adjust our view:
lists/views.py.
from django.shortcuts import render
from lists.models import Item
def home_page(request):
item = Item()
item.text = request.POST.get('item_text', '')
item.save()
return render(request, 'home.html', {
'new_item_text': request.POST.get('item_text', ''),
})
I’ve coded a very naive solution and you can probably spot a very obvious problem, which is that we’re going to be saving empty items with every request to the home page. Let’s add that to our list of things to fix later. You know, along with the painfully obvious fact that we currently have no way at all of having different lists for different people. That we’ll keep ignoring for now.
Remember, I’m not saying you should always ignore glaring problems like this in "real life". Whenever we spot problems in advance, there’s a judgement call to make over whether to stop what you’re doing and start again, or leave them until later. Sometimes finishing off what you’re doing is still worth it, and sometimes the problem may be so major as to warrant a stop and rethink.
Let’s see how the unit tests get on … they pass! Good. We can do a bit of refactoring:
lists/views.py.
return render(request, 'home.html', {
'new_item_text': item.text
})
Let’s have a little look at our scratchpad. I’ve added a couple of the other things that are on our mind:
Let’s start with the first one. We could tack on an assertion to an existing test, but it’s best to keep unit tests to testing one thing at a time, so let’s add a new one:
lists/tests.py.
class HomePageTest(TestCase):
[...]
def test_home_page_only_saves_items_when_necessary(self):
request = HttpRequest()
home_page(request)
self.assertEqual(Item.objects.count(), 0)
That gives us a 1 != 0 failure. Let’s fix it. Watch out; although it’s quite a small change to the logic of the view, there are quite a few little tweaks to the implementation in code: